1. Field of the Invention
This invention relates to hardware implementation of artificial neuron, specifically to a synapse element, a threshold circuit or a neuron circuit using the synapse elements, and a learnable neuron device.
2. Description of the Related Art
The highly developed Neumann-type computer has a far better performance in numerical calculation compared with the human ability. This type of computer still requires vast calculating time in pattern recognition or image processing, and it is insufficient especially in information processing such as association, memorization and learning which are easily performed by a human brain. There is an approach to utilize neural networks constructed by simulating functions of the human brain for providing the computer calculating facilities to perform the above processings.
The human brain has highly sophisticated functions though its basic construction is rather simple. The brain consists of nerve cells called neurons having calculating functions and nerve fibers propagating the calculated results to the other neurons. A neuron connects to nerve fibers by so-called synapse links, and the synapse links provide the neuron with the signals propagated through each nerve fiber after modifying them each with a proper weight function. The neuron is stimulated to provide an output signal to the nerve fibers connected to other neurons, which phenomenon is called fire, only when the total sum of the signals input through the nerve fibers exceeds a certain threshold value, while the neuron does not produce any output signal when the total sum is lower than the threshold value.
The human brain having a six-layer structure of a vast number of neurons connected each other performs complex information processings. The human learning process can be recognized as a process for changing the weights in the synapse links.
The neural networks are calculation structures constructed by complied layers in multiple stages formed of a vast number of elements having neural function connected each other modeling the nervous network of human brain. Therefore, the neural networks are expected to perform more easily high-degree processings of information such as pattern recognition, image processing, association, memorization, learning, and so on.
The development of semiconductor technology in recent years brings possibility to provide physical neural networks based on their conceptional designs.
Today""s and especially future networks require or can require hundreds or even thousands of neurons with hundreds of input synapses each. Saving just one transistor saves tens and hundreds of thousands transistors.
A threshold element (TE) has been commonly studied because it is the simplest model of the neuron. A threshold element is proposed to be constructed by xcexdMOS circuits having analogue amplifiers, output wired inverters, or floating gates.
The critical parameter is the permissible sum of the input weights and threshold which depends on possible variations of technological and physical parameters. In the learnable neurons, the parameter variations are compensated during the learning and the critical parameter becomes the sensitivity of the output amplifier that, in fact, is of the same order for most available implementations. Hence, the main criterion for choosing the basic TE when implementing a learnable artificial neuron should be the number of transistors per one synapse.
In the conventional semiconductor technologies, a vast number of semi-conductors should be combined to realize the functions in only one neuron. Even a limited number of semiconductors required in a restricted practical use cannot be integrated in a single semiconductor chip, so that construction of practical neural networks is a sufficiently difficult technical task.
JPA03-006679 discloses an invention for integrating the functions of one neuron onto one MOS (metal-oxide semiconductor structure) transistor so as to solve the above problem.
The above disclosed semiconductor device is a MOS semiconductor element called xcexdMOS with a floating gate and a plural number of capacitance coupling input gates. Each of the input gates is equipped with an electrode having a proper area corresponding to its input weight. The input voltage provided to the input gate is multiplied with the proper weight determined by the electrode area. And the sum of the input voltages provided to the relating input gates corresponds to the voltage of the floating gate. When the sum exceeds a certain threshold level, a channel is formed under the floating gate electrode and an electric current runs through the channel as corresponding to the neuron""s fire.
xcexdMOS enables to decrease the transistor area in the semiconductor chip by tenth order as the number of the required bipolar transistors is decreased, and to realize low power consumption as MOS transistors are voltage-controlled devices. Thus the above approach is expected to obtain a real neuron computer.
According to the above technology in which the weights are determined by the capacitance coupling of the input gates to the floating gate, the number of the inputs and the weights to be applied to the inputs cannot change after building the device, because they are fixed by the structure of the xcexdMOS. Therefore, the device is impossible to simulate flexible ability of the human brain relating information processings as recognition, association, learning, and so on. A method for solving the problem is proposed in which the device is furnished with multipliers for adjusting parameters and the adjusted parameters are applied to the input signals and then the input signals are provided to the xcexdMOS. The device applied with the above method has much flexibility in functions though it needs more semiconductors per one synapse.
JPA6-139380 discloses a xcexdMOS furnished with a self-learning facility. The disclosed device adjusts charges in the floating gate when the calculated value differs from the indicated value so as to adjust the weight of the synapse. This device can automatically adjust the weight function of the synapse which provides signals to the neuron, thus it can eliminate the need of an outstanding computer to calculate the weight of each synapse and may obtain an optimal control of the system through learning on the spot. Simulations have proven a high degree optimization in some logics to be executed.
JPA10-54079 discloses a xcex2-driven threshold element (xcex2DTE) invented by the inventor of this invention.
Any linear logics and some kind of logics can be represented in the threshold function below:
Y=Sign(xcexa3j=0xcx9cnxe2x88x921xcexa9jxjxe2x88x92T)=Sign(xcexa3jxcex5!sxcexa9jxjxe2x88x92xcexa3jxcex5sxcexa9j!xj)
where S is a subset of variables such that xcexa3jxcex5sxcexa9j=T, ! means negation, Xj is 0 or 1, and xcexa9 is an integer.
xcexa9 j is normalized with T to obtain xcfx89j=xcexa9j/T. Parallel connected p-channel MOS transistors whose input xj belongs to a certain subset S and its xcex2-value is adjusted to the corresponding xcfx89j are serially connected with parallel connected n-channel MOS transistors whose input does not belong to the subset S and its b-value is adjusted to the corresponding xcfx89j. Then the output voltage Vout at the terminal connecting point is indicated as follows:
vout=xcexa3jxcex5!sxcfx89jxj/xcexa3jxcex5sxcfx89j!xjxe2x80x83xe2x80x83(1)
Therefore, a comparator inverter easily determines truth of the threshold function by comparing the output Vout with threshold value a, which is set as follows:
xcex1=xcexa3jxcex5!sxcfx89j/xcexa3jxcex5sxcfx89j!=xcexa3xcex2n/xcexa3xcex2pxe2x80x83xe2x80x83(2).
The xcex2DTE is constructed according to the above principle. FIG. 13 is a circuit diagram indicating a principle of the xcex2DTE.
Variables Xj belonging to the subset S in n number of Xj""s described in the logics are provided to p-channel MOS transistors P1, P2, . . . Pk. Variables Xj which are not belong to the subset S are provided to n-channel MOS transistors N1, N2, . . . Nnxe2x88x92k. The drain of the whole transistors are connected each other. The voltage Vout appearing at the connected drain is provided to the comparator inverter having a threshold value of a, and the output from the comparator inverter represents the final result Y.
This type of xcex2DTE needs one MOS transistor corresponding to one input, and provides a threshold element consisting of a least number of transistors. The current amplification factor xcex2 is determined by length and width of the gate electrode, so that the input weight function xcfx89j can be determined by width of the transistor.
The brain has a main advantage in learning ability, therefore if an artificial neural network has not learning function the network cannot sufficiently simulate the information processing function of the human brain.
The above described conventional arts have some problems such that the input weights are to be fixed without learning functions because the artificial neuron is built up using a special structure of semiconductors and cannot vary the weights after completion, or that the device should furnish with an excessive equipment as multipliers for adding learning facility although the device itself may be a simple synapse or a simple neuron.
It is therefore an object of this invention to provide a synapse element, a neuron circuit and a neuron network, and especially a learnable neuron device, consisting of a smaller number of elements utilizing the commonly used semiconductor technologies.
Other object of the invention will appear in the course of the description thereof which follows.
To achieve the foregoing objects, a synapse element of the invention comprises a transistor set made of the first MIS (metal insulator semiconductor) transistor having the first gate electrode and the second MIS transistor having the second gate electrode. The second MIS transistor is connected in series with the first MIS transistor. The voltage of the first input signal provided to the first gate electrode adjusts the effective b-value of the transistor set. And the transistor set is switched according to the second input signal provided to the second gate electrode.
In place of the transistor set having two transistors, a single transistor having two separated electrodes may be used in which the effective b-value of the single transistor can be adjusted by the input voltage applied to the first electrode.
The synapse of the invention preferably includes a voltage holding element connected with the first gate electrode, and a switch element between the first gate electrode and terminals each connected to a high voltage power source and to a low voltage power source, in which the switch element is switched according to the second input signal.
The switch element may be a MIS transistor receiving the second input signal by its gate electrode, and the voltage holding element may be a capacitance element.
In the synapse element of the invention, the effective b-value is adjusted by the first transistor and turning on and off is controlled by the signal input to the second transistor, therefore the input weight of the synapse can be easily selected and fixed by adjusting the effective b-value of the transistor. According to the invention, an element simulating a synapse can be constructed which generates an output signal applied with a weight to an input signal propagated through a nerve fiber.
The synapse element of the invention can be constructed by MIS transistor elements of usual type because the synapse element, unlike the conventional synapse element, does not require any special construction of the gate electrode or any adjustment of geometrical sizes of the respective transistors.
The synapse element of the invention requires very few MIS transistors, therefore the synapse element can be easily constructed on a single substrate and a plurality of the synapse elements may be integrated in a smaller area of the substrate.
In a learnable neuron device composing of the synapse elements having voltage holding elements such as capacitors and switching elements using MIS transistors thereby maintaining the effective b-value by holding the voltage applied on the gate electrode and turning the switching elements on and off according to the input signal applied on the second transistors, appropriate synapse elements can be easily selected to be varied with their weights on learning and the voltage applied on the selected elements can be adjusted in the proper direction.
If the elements are composed of MIS transistors, the whole device can be formed on a single substrate so that a very small device integrated with a vast number of synapse elements may be provided.
To achieve the foregoing objects, a threshold circuit of the invention is characterized in comprising any one of the above described synapse elements connected in parallel whose second transistors have input terminals, a common line connected with the first connecting terminals of the synapse elements connected in parallel, a power line connected with the second connecting terminals of the synapse elements through a third MIS transistor having a complementary channel type, and an output inverter connected with the second connecting terminals by the input terminal of the inverter.
It is preferable that the MIS transistors of the synapse elements are n-channel MOS (metal-oxide semiconductor) transistors and the third MIS transistor is a p-channel MOS transistor.
A second threshold circuit of the invention comprises a positive power line connected to a positive electrode of a power source, a negative power line connected to a negative electrode of the power source, a plurality of n-channel transistor sets each composing of a first n-channel MIS transistor and a second n-channel MIS transistor connected in series with the first MIS transistor, at least one p-channel MIS transistor, and an output inverter, wherein the first n-channel MIS transistor adjusts an effective xcex2-value of the transistor set according to the voltage applied on the gate electrode of the first n-channel MIS transistor and the second n-channel MIS transistor switches the transistor set by applying an input signal on the gate electrode of the first MIS transistor. In addition, the positive power line is connected with the source electrode of the p-channel MIS transistor and the negative power line is connected with the gate electrode of the transistor, the plurality of the n-channel transistor sets are connected in parallel with each other, the source electrode terminals of the parallel connected n-channel transistor sets are connected with the negative power line, and the drain electrode terminals of the transistor sets are connected to the drain electrode of the p-channel MIS transistor, and the drain electrode terminal of the p-channel transistor is connected to the input terminal of the output inverter.
The threshold circuit of the invention simulates a neuron function element which determines if the sum of input signals each applied with an arbitrary weight exceeds a certain threshold value, wherein the weight is determined by adjusting xcex2-values of synapse elements connected in parallel and the synapse elements are corresponding to an arbitrary number of input signals, respectively.
The main components of the threshold circuit are MIS transistors and the number of the transistors used in the circuit is small, therefore many circuits are easily integrated in a narrow area on a semiconductor substrate.
The threshold circuits of the invention are applied to artificial neuron networks, majority logic circuits, filtering circuits, and so on.
Any of the above mentioned threshold circuits can be constructed with synapse elements each having an effective xcex2-value corresponding to a weight xcfx89i derived from a logical equation Y=sign(xcexa3xcfx89iXixe2x88x921) transformed from a logical equation Y=F(Xi), and a comparing element compares a combined output signal with a threshold value and provides a comparison result.
Small and economic integrated circuits for deciding truth of logical equations are obtained with very small number of usual semiconductors by the above-described method to set the effective xcex2-values.
And a neuron device of the invention, to achieve the foregoing objects, comprises a threshold circuit composed of a plurality of synapse elements each of which is adjusted with its effective xcex2-value from the outside and generates an output signal controlled by the xcex2-value upon receiving an input signal, an input signal generator for providing corresponding input signals to each input terminal of the synapse elements, a truth table circuit for providing expected values of output signals corresponding to any given set of input signals, a comparator for comparing the output signals of the threshold circuit with the expected values of the output signals, adjusting device for generating increment signals or decrement signals selectively based on the output signals of the comparator, wherein the adjusting device and a xcex2-value adjusting terminal of the synapse element are connected only when the synapse is stimulated by the input signal.
It is preferable that the threshold circuits of the neuron device of the invention are the threshold circuits of the invention described above.
A second neuron device of the invention comprises a first current-control element, a second current-control element which is connected with the first current-control element in series laying between a first voltage and a second voltage, an auxiliary circuit for varying characteristic factor of at least the second current-control element and providing the input signal to at least one of the second current-control elements, and a threshold deciding circuit for receiving an intermediate voltage which appears at the connection point of the first current-control element and the second current-control element, wherein the auxiliary circuit controls the intermediate voltage corresponding to even a same set of the input signals so as to adjust results of threshold decision process.
The first current-control element may be constantly conductive, while the second current-control element is controlled on and off by the input signal applied to the element.
A plurality of the second current-control elements may be connected in series so as to receive a plurality of input signals.
It is preferable that the first current-control element and the second current-control element are complementary field-effect transistors to each other.
A current amplification factor may be used as the adjustable characteristic factor of the current-control elements.
The auxiliary circuit may comprise a voltage holding element connected with the gate electrode of the field-effect transistor and a switching element for controlling an electric current charging or discharging the voltage holding element, wherein the switching element is opened or closed according to the input signal.
It is preferable that the second current-control elements are connected in parallel, and the neuron device further comprises an adjusting device for controlling the auxiliary circuit according to the comparison result of the threshold decision and the expected values so that the neuron device furnishes a learning ability.
The neuron device of the invention provides a neuron network having very few semiconductor elements which can adjust the input weights of the synapses by comparing the output results with the expected values and can display a learning ability resembling human ability by maintaining the adjustment results.